The Effect of Rating Scales on Decision Quality and User Attitudes in Online Innovation Communities

Given the rise of Internet, consumers increasingly engage in co-creating products and services. Whereas most co-creation research deals with various aspects of generating user-generated content, this study addresses designing ratings scales for evaluating such content. In detail, we analyze functional and perceptional aspects of two frequently used rating scales in online innovation communities. Using a multi-method approach, our experimental results show that a multi-criteria scale leads to higher decision quality of users than a single-criteria scale, that idea elaboration (i.e., idea length) negatively moderates this effect such that the single-criteria rating scale outperforms the multi-criteria scale for long ideas, and finally that the multi-criteria scale leads to more favorable user attitudes towards the website. Based on our experimental data, we applied a bootstrap-based Monte Carlo simulation for assuring robustness of our results. We find that around 20 user ratings per idea are sufficient for creating stable idea rankings and that a combination of both rating scales leads to a 63% performance improvement over the single-criteria rating scale and 16% over the multi-criteria rating scale. Our work contributes to co-creation research by offering insights as to how the interaction of the technology being used (i.e., rating scale), and attributes of the rating object affects two central outcome measures: the effectiveness of the rating in terms of decision quality of its users and the perception of the scale by is users as a predictor of future use.